Consider a large multi-user computer system which maintains a cumulative journal of all transactions executed by the computer system (sometimes called the log file). Typically, each log entry will identify the transaction, indicate whether the transaction was completed or aborted, and include other relevant data. Over time, a very large number of such log entries will accumulate, and these entries are typically stored in sequential disk files. In a complex transaction handling computer system with long-lived transactions and a large, continuing load of transactions, log records are created at a very high rate, requiring insertion of log records into a file at high frequency.
In at least some situations, later access to log records by one or more indexed values is desirable. For instance, some system users may access such historical records with sufficient frequency that the files need to be stored in a specialized database management system (DBMS) using indexed references in order to enable efficient access to this data.
In many transaction handling computer systems, logging has been focused on assuring recovery from system failures, and thus has required the system to be able to refer back to a relatively short-term history of events. However, as system loads continue to grow and as transactional systems take on responsibility for more complex activities, the requirement for activity recording (event logging) shifts from simple recovery logging to long-term activity flow management support, system monitoring, security, and so on. With this shift, the duration and number of events which make up a single long-lived transaction will increase to the point where there is a frequent need for human agents to review the steps taken up to a given time. At the same time, the total number of active events known to a system will increase to the point were memory-resident data structures now used to keep track of active events are no longer feasible.